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INTRODUCTION  AND  EXECUTIVE  SUMMARY 
Research  Goals 

Injured  Warfighters  return  home  to  face  long-term  care  and  recovery  in  addition  to  life  adjustments.  Rehabilitation 
services — such  as  interventions  for  traumatic  brain  injury  (TBI)-induced  motor  limitations,  broken  bones,  spinal  cord 
injuries,  chronic  pain,  and  amputation — enable  these  Warfighters  to  adjust  to  new  living  constraints  and  conditions  and, 
in  many  cases,  return  to  full  health.  While  these  services  are  readily  available  in  military  treatment  facilities  (MTFs)  and 
veterans’  affairs  medical  centers  (VAMCs),  not  all  patients  have  the  time  or  ability  to  receive  prolonged  inpatient 
rehabilitation  interventions.  Furthermore,  lengthy  inpatient  treatments  are  costly  to  MTFs  and  VAMCs,  reducing  the 
overall  number  and  types  of  services  that  these  facilities  can  provide. 

Home -based  and  outpatient  rehabilitation  interventions  hold  great  potential  to  improve  the  rehabilitation  of  our 
Warfighters.  These  rehabilitation  interventions  enable  patients  to  continue  with  their  daily  lives  during  rehabilitation. 
Patients  can  perform  professional  duties;  be  with  family;  and  be  social  with  friends  around  the  schedule  of  their 
rehabilitation  practice,  and  all  of  these  functions  enable  patients  to  better  adjust  to  life  changes  that  follow  injury.  Home- 
based  and  outpatient  rehabilitation  interventions  are  accessible  to  a  wide  range  of  patients  because  they  lower  the  time 
and  travel  requirements  of  rehabilitation.  Finally,  home -based  and  outpatient  rehabilitation  interventions  are  less  costly 
to  MTF  and  VAMC  service  providers,  enabling  these  facilities  to  provide  a  wider  range  of  services  to  more  patients. 

The  patient  must  practice  therapeutic  exercises  regularly.  The  unfortunate  reality  of  many  home -based  and  outpatient 
therapies  is  that  the  patient  does  not  regularly  practice  therapeutic  exercises  beyond  visits  with  the  therapist  and, 
therefore,  does  not  see  significant  improvement.  Studies  of  home -based  and  outpatient  rehabilitation  interventions  have 
identified  a  number  of  key  correlates  to  lack  of  adherence:  confusion  about  exercises;  perceptions  of  lack  of  time  to 
exercise;  forgetting  to  exercise;  perceptions  of  helplessness;  and  overall  lack  of  motivation  to  exercise  (Jette  et  al.,  1998; 
Sluijs,  Kok,  &  van  der  Zee,  1993).  Conversely,  patients  who  have  less  confusion,  make  time  to  exercise,  remember  to 
exercise,  perceive  higher  self-efficacy,  and  report  motivation  to  exercise  adhere  more  regularly  to  rehabilitation 
protocols.  In  addition  to  these  areas  of  needed  patient  assistance,  outpatient  therapists  must  be  enabled  to  perform  their 
job  functions  of  observing  the  patient  and  directing  exercises. 

For  these  reasons,  remote  assistance  to  ho  me -based  and  outpatient  rehabilitation  is  needed  to  enhance  the  recovery 
of  our  injured  Warfighters. 

Description  of  the  Technical  Approach 

To  address  these  issues,  we  are  developing  mobile,  virtual  enhancements  for  rehabilitation  (MOVER),  a  mobile, 
technology-enabled  home-based  rehabilitation  intervention  delivery  system.  MOVER  features  (1)  a  mobile 
application  to  provide  education,  information,  and  scheduling  of  therapeutic  exercises;  (2)  virtual  coaches  to  guide, 
mentor,  and  motivate  patients;  (3)  COTS  input  devices  and  video  games  to  increase  patient  motivation;  and  (4)  a  web- 
based  therapist  interface  to  accurately  assess  patient  adherence  and  progress. 

Figure  1  shows  the  MOVER  Architecture.  At  the  top  left  of  the  figure,  the  Patient  interacts  with  the  MOVER  Mobile 
Application  to  perform  Exercise  Scheduling  and  obtain  Information  and  Education  about  therapeutic  exercises.  The 
Virtual  Coaches  exist  on  the  mobile  application  and  provide  interactive  guidance  and  mentoring  about  the  rehabilitation 
process  and  therapeutic  exercises.  When  the  scheduled  time  for  the  exercises  arrives,  the  mobile  application  reminds  the 
patient,  and  the  patient  begins  an  exercise  session  with  the  MOVER  Game  Integration,  as  shown  at  the  bottom  of  the 
figure.  The  patient  uses  COTS  input  devices,  such  as  the  Microsoft  Kinect  and  the  Wii  Balance  Board,  to  perform 
therapeutic  exercises  that  are  mapped  to  controls  of  the  Video  Game  Console  through  the  Control  Mapping  on  Laptop, 
software  running  on  an  inexpensive  PC  or  laptop. 

During  interaction  with  the  mobile  application  and  game  integration,  patient  Performance  is  recorded  and  sent  securely  to 
the  Remote  Server  and  Secure  Database,  at  the  center  of  the  figure.  The  Therapist  reviews  this  performance  through 
summarized  Progress  Reports  in  the  web-based  Therapist  Interface,  as  shown  at  the  top  right  of  the  figure.  The  therapist 
then  creates  Therapeutic  Exercise  Assignments  to  describe  the  patient’s  therapeutic  exercises  for  the  next  week,  and  these 
assignments  are  passed  to  the  game  integration  for  implementation  the  next  time  the  patient  begins  exercise. 
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Figure  1:  MOVER  architecture 

A  typical  use  of  MOVER  in  home -based  rehabilitation  is  as  follows.  The  patient  meets  with  the  therapist  at  the 
beginning  of  the  week  to  participate  in  a  short,  one-on-one  rehabilitation  session.  The  therapist  assesses  the 
patient,  prescribes  a  set  of  therapeutic  exercises  for  the  week,  and  works  with  the  patient  to  determine  a  feasible 
exercise  schedule.  At  the  end  of  the  session,  the  patient  and  therapist  enter  an  exercise  schedule  for  the  week,  and 
the  therapist  enters  the  therapeutic  exercises  into  the  therapist  interface.  During  the  week,  the  patient  is  reminded 
of  scheduled  exercises  by  the  mobile  application  and  motivated  by  the  virtual  coaches.  The  patient  uses  the  COTS 
input  devices  to  play  the  video  games  with  the  therapeutic  movements  specified  by  the  therapist.  During  exercise, 
the  virtual  coaches  give  feedback  on  patient  movements  and  form,  and  afterwards  the  virtual  coaches  review  the 
patient  performance.  Performance  information  is  collected  and  sent  to  the  remote  server.  At  the  next  session,  the 
therapist  reviews  the  progress  reports  with  the  patient  to  determine  next  steps  for  treatment. 
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Figure  2:  MOVER  deliverables  and  milestones  schedule 


TECHNICAL  PROGRESS 
Progress  against  Planned  Objectives 

During  the  current  reporting  period,  we  began  our  initial  requirements  analysis  and  development  as  per  the 
schedule  in  Figure  2. 

Technical  Accomplishments  This  Period 

During  this  reporting  period,  we  focused  on  Task  1  (Requirements  Analysis),  Task  2  (Mobile  Application 
Development),  Task  3  (Virtual  Coaches  Development),  Task  4  (Input  Device  and  Game  Integration),  and  Task 
7(Program  Management)  as  presented  in  the  Statement  of  Work  for  this  effort. 

Task  1:  Requirements  Analysis 

Our  goal  for  Task  1  is  to  conduct  knowledge  elicitation  with  TB1  motor  impairment  therapists  and  patients, 
identify  needed  extensions  to  existing  treatments,  and  create  a  requirements  specification  for  this  system. 

During  the  current  reporting  period,  we  (1)  began  our  discussions  with  TBl-induced  mobility  limitation  subject 
matter  experts  (SMEs),  including  both  therapists  and  researchers  who  interact  with  this  patient  population  (2) 
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developed  user  personas  to  represent  potential  users  of  the  system  and  (3)  clarified  initial  requirements  of  the 
system. 

Through  SME  discussions,  we  identified  additional  context  for  the  use  of  the  system.  Once  MOVER  is  fully 
developed,  we  envision  that  patients  will  be  interacting  with  the  games  in  an  outpatient  or  self-directed  setting 
within  the  clinic  (e.g.,  have  the  system  available  in  their  inpatient  room).  Because  this  population  may  have 
balance  control  issues  that  may  lead  to  falls  or  injuries,  patients  must  be  carefully  screened  in  these  cases  to 
determine  the  safety  of  playing  the  games  unsupervised.  Also,  some  exercises  may  be  performed  from  non¬ 
standing  positions,  greatly  reducing  the  likelihood  of  injuries.  These  include  both  seated  exercises  for  vestibular 
problems  that  may  lead  to  gaze  instability  and  general  strengthening  and  conditioning  exercises  such  as  lying 
down  leg  and  upper  extremity  exercises.  We  may  use  these  limitations  as  criteria  for  our  evaluation  study  (under 
the  Task  6  Option),  though  we  plan  to  supervise  participants  closely  during  this  stage  of  development.  One  of  the 
objectives  of  the  evaluation  will  be  to  determine  which  populations  may  safely  use  MOVER  unsupervised. 

We  have  tentatively  set  the  date  of  January  14th  2014,  as  our  next  on-site  visit  at  Spaulding  Rehabilitation 
Hospital.  At  this  meeting  we  will  hold  knowledge  elicitation  with  our  therapist  SMEs  in  the  morning  and  observe 
patient  therapy  sessions  in  the  afternoon.  We  will  finalize  this  date  once  we  receive  final  approval  from  Spaulding 
management  for  the  patient  observations  and  therapist  schedules  are  set  for  the  new  year. 

To  better  meet  the  needs  of  our  users,  we  have  developed  several  user  personas  that  represent  the  needs  and 
variations  of  potential  users  of  the  system.  Grudin  and  Pruitt  (2002)  note  about  persona  design:  “Not  only  can  it  be 
a  powerful  tool  for  true  participation  in  design,  it  also  forces  designers  to  consider  social  and  political  aspects  of 
design  that  otherwise  often  go  unexamined.”  The  personas  we  have  developed  to  date  include  3  patients,  2 
therapists,  a  therapist  group  leader,  and  a  hospital  information  technology  (IT)  administrator.  Through  this  process 
we  identified  several  key  variants  in  patients,  including: 

•  Condition  and  physical  capability 

•  Technical  savviness 

•  Video  game  experience  and  preferences 

•  Other  TBI  symptoms 

•  Comorbid  health  and  mental  health  issues  to  TBI  (e.g.,  PTSD) 

•  Attitude  toward  treatment 

•  First  language  and  general  reading  ability 

•  Ability  to  attend  therapy 

•  Drive  and  determination  to  improve  abilities 

We  also  identified  key  variants  in  therapists  and  staff: 

•  Job  experience 

•  Knowledge  of  TBI  and  related  therapies 

•  Experience  with  inpatient  and  outpatient  settings 

•  Emotional  support 

•  Technical  savviness 

•  Willingness  to  try  new  therapies 

•  Cost  concerns 

In  addition  to  our  design  and  SME  interactions,  we  attended  the  Artificial  Intelligence  and  Interactive  Digital 
Entertainment  (AIIDE)  2013  conference  in  Boston,  MA  to  stay  abreast  of  the  latest  developments  in  this  research 
area  and  make  key  contacts  for  future  MOVER  development.  AIIDE  is  focused  on  the  latest  advances  in  AI  as 
they  relate  to  interactive  digital  experiences,  and  it  is  therefore  highly  related  to  the  goals  of  MOVER.  In  addition 
to  numerous  interactions  with  university  and  private  sector  researchers  in  this  area,  many  of  the  conference  papers 
and  talks  were  informative  for  MOVER  development,  especially  for  the  virtual  coaches  and  COTS  games  use. 
Below  are  brief  summaries  of  three  of  these  relevant  papers  and  talks. 
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Sara  Bernardini,  Kaska  Porayska-Pomsta,  Harini  Sampath,  Designing  an  Intelligent  Virtual  Agent  for  Social 
Communication  in  Autism,  A11DE,  Boston,  MA.  2013. 

From  the  abstract:  “This  paper  describes  the  Intelligent  Engine  (IE)  of  ECEIOES,  a  serious  game  built  for  helping 
young  children  with  Autism  Spectrum  Conditions  acquire  social  communication  skills.  ECEIOES  IE’s  main 
component  is  an  autonomous  virtual  agent  that  acts  as  a  credible  social  partner  for  children  with  autism  by 
engaging  them  in  interactive  learning  activities.” 

This  virtual  agent  is  based  on  previous  work  in  the  FAtiMA  agent  architecture  (Dias  and  Paiva  2005;  Aylett,  Dias, 
and  Paiva  2006).  FAtiMA  agents  use  a  theory  of  emotions  and  appraisal  theory  to  interpret  events  and  generate 
expressive  behavior.  These  agents  appraise  the  environment  around  them  through  scripted  rules  and  then  cope 
with  the  environment  and  their  internal  emotions  using  either  problem-focused  (e.g.,  problem  solving)  or  emotion- 
focused  (e.g.,  expression  or  re-interpretation)  strategies.  These  agents  have  been  shown  to  be  engaging  for 
straightforward  tasks  that  involve  an  emotional  component.  With  the  MOVER  virtual  coaches,  these  strategies  of 
appraisal  and  coping  can  be  incorporated  to  provide  more  authentic  and  engaging  interaction. 


Andrew  Kope,  Caroline  Rose,  Michael  Katchabaw,  Modeling  Autobiographical  Memory  for  Believable  Agents, 
A1IDE,  Boston,  MA.  2013. 

From  the  abstract:  “We  present  a  multi-layer  hierarchical  connectionist  network  model  for  simulating  human 
autobiographical  memory  in  believable  agents.  Grounded  in  psychological  theory,  this  model  improves  on 
previous  attempts  to  model  agents’  event  knowledge  by  providing  a  more  dynamic  and  nondeterministic 
representation  of  autobiographical  memories.” 

This  paper  models  memory  of  world  events  and  objects  in  virtual  agents  by  an  ACT-R  style  connectionist 
network.  This  network  enables  the  virtual  characters  to  recall  experiences  and  associations  in  a  more  naturalistic 
and  believable  manner.  For  the  MOVER  virtual  coaches,  the  recall  of  specific  interactions  with  a  patient  and 
natural  associations  between  these  interactions  can  be  a  useful  tool  for  building  history  and  rapport  with  the 
patient,  as  well  as  motivating  exercises.  The  basic  assumption  is  that  if  the  coach  exhibits  memories  of 
interactions,  these  interactions  become  more  meaningful  and  the  patient  will  treat  them  with  more  care. 


Daniel  Shapiro,  Josh  McCoy,  April  Grow,  Ben  Samuel,  Andrew  Stern,  Reid  Swanson,  Mike  Treanor,  Michael 
Mateas,  Creating  Playable  Social  Experiences  through  Whole-Body  Interaction  with  Virtual  Characters,  AIIDE, 
Boston,  MA.  2013. 

From  the  abstract:  “This  paper  describes  work  towards  the  goal  of  enabling  unscripted  interaction  with  non-player 
characters  in  virtual  environments.  We  hypothesize  that  we  can  define  a  layer  of  social  affordances,  based  on 
physical  and  non-verbal  signals  exchanged  between  individuals  and  groups,  which  can  be  reused  across  games.” 

This  paper  presents  emerging  work  combining  previous  work  on  social  simulations  with  gestures  and  whole-body 
interactions,  enabling  virtual  agents  to  be  socially  aware  of  their  own  and  the  player’s  non-verbal  signals.  The 
MOVER  system  has  the  ability  to  track  patient  movements  and  may  be  able  to  discern  key  social  indicators  that 
could  inform  interaction  with  the  system.  Also,  the  body  language  and  social  cues  of  the  virtual  coaches  are  a 
powerful  feedback  channel  and  motivator  for  the  patient  which  may  be  used  via  models  of  non-verbal 
communication  such  as  the  one  described  in  this  paper. 

Task  2:  Mobile  Application  Development 

Our  goal  for  Task  2  is  to  develop  a  mobile  application  to  accompany  home-based  TBI  motor  impairment 
rehabilitation  therapy.  We  will  also  enhance  therapy  protocols  to  use  the  mobile  application  and  develop  methods 
to  enable  therapist  and  peer  support  through  the  mobile  application. 

During  the  current  reporting  period,  we  investigated  the  use  of  our  current  iOS-based  application  framework  for 
developing  the  MOVER  mobile  application  (app).  The  framework  is  flexible  and  highly  usable  for  developing 
apps  that  consist  largely  of  static  screens  with  input  forms  and  multimedia  such  as  pictures,  sounds,  and  videos.  It 
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supports  sending,  storing,  and  receiving  data  securely  with  a  remote  server,  which  MOVER  may  use  to  provide  a 
therapist  portal.  Because  of  these  features  and  our  existing  expertise  in  this  framework,  it  will  be  a  strong  platform 
to  begin  our  development.  The  to-be-defined  requirements  of  the  virtual  coaches  my  necessitate  expanding  this 
framework  to  include  technologies  that  allow  for  real-time  animation  and  interaction  with  virtual  characters. 
Possibilities  for  these  capabilities  largely  include  game  engines  such  as  Microsoft’s  XNA  framework  or  the  Unity 
3D  platform. 

Task  3:  Virtual  Coaches  Development 

Our  goal  for  Task  3  is  to  develop  virtual  coaches  to  assist  home-based  TB1  motor  impairment  rehabilitation 
therapy  through  mentoring  and  guidance.  The  virtual  coaches  development  is  awaiting  finalization  of  the  virtual 
coaches  requirements.  The  virtual  coaches  are  one  of  the  most  flexible  components  of  the  MOVER  system  and 
will  be  highly  tailored  to  the  needs  of  the  end  users. 


Task  4:  Input  Device  and  Game  Integration 

Our  goal  for  Task  4  is  to  integrate  commercial-off-the-shelf  (COTS)  input  devices  and  COTS  video  games  to 
engage  the  patient  and  estimate  body  posture  during  therapeutic  exercises. 


During  the  current  reporting  period,  we  investigated  potential  input  devices.  Patients  interact  with  the  video  games 
through  COTS  movement-based  input  devices,  such  as  the  Microsoft  Kinect  or  Nintendo  Wii  Balance  Board.  The 
input  devices  are  connected  to  a  computer  laptop  running  MOVER  gesture  recognition  algorithms  that  remap 
patient  movements  to  game  controls  that  are  sent  to  the  video  game  console.  The  current  generation  Kinect  has 
several  APIs  and  open  toolkits  available,  including  the  official  Microsoft  SDK.  We  have  integrated  with  these 
APIs  under  previous  efforts,  as  shown  in  Figure  3.  Microsoft  has  announced  the  next  generation  of  the  Kinect 
sensor  will  include  higher  definition  video,  3D,  and  audio  tracking,  scheduled  for  release  in  summer  of  2014.  The 
additional  resolution  and  tracking  capability  of  this  new  sensor  would  be  extremely  beneficial  to  MOVER  as  it 
monitors  patient  movements  during  exercise.  We  will  continue  to  follow  this  progress  and  investigate  early 
developer  programs  to  move  towards  obtain  this  technology. 


Figure  3:  Microsoft  Kinect  input  creating  a  3D  representation  of  user  poses 

In  addition  to  the  Kinect  sensor,  the  Nintendo  Wii  balance  board  and  the  new  Sony  Playstation  4  Eye  camera  offer 
potential  alternatives  to  monitor  patient  movement.  Specific  tradeoffs  between  these  devices  include  range  and 
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type  of  detectable  motion,  accuracy,  and  cost.  We  will  make  this  determination  for  our  initial  MOVER  prototype 
as  we  finalize  our  system  requirements,  while  maintaining  the  possibility  for  rapidly  including  new  and  additional 
input  devices  through  well-defined  APIs. 

Task  7:  Program  Management 

Our  goal  for  Task  7  is  to  coordinate  team  activities,  author  status  reports,  and  brief  the  Sponsor  on  project  progress 
as  appropriate. 

During  the  current  reporting  period,  we  held  the  kickoff  briefing  for  MOVER  at  ONR  offices  with  LCDR  Brent 
Olde  and  Dr.  Paul  Chatelier. 

We  have  executed  the  subcontract  agreement  and  Spaulding  Rehabilitation  Hospital  is  now  under  contract  for 
MOVER.  We  also  finalized  and  received  approval  for  our  evaluation  from  the  Spaulding  Rehabilitation  Hospital 
IRB. 

Significant  Changes  to  Technical  Approach  to  Date 

At  the  time  of  writing,  no  significant  changes  have  been  made  to  the  MOVER  technical  approach. 

Deliverables  Submitted  This  Period 

No  deliverables  were  submitted  this  reporting  period. 

Milestones  Reached/Achieved  During  This  Period 

No  milestones  were  scheduled  for  this  reporting  period. 


PROJECT  PLANS 

Specific  Objectives  for  Next  Period 

■  During  the  next  time  period,  we  plan  to  focus  on  Task  1  (Requirements  Analysis), Task  2  (Mobile 
Application  Development),  Task  3  (Virtual  Coaches  Development),  Task  4  (Input  Device  and  Game 
Integration),  Task  5  (Remote  Server  and  Therapist  Interface  Development),  and  Task  7(Program 
Management). 

■  Under  Task  1  we  will  conduct  knowledge  elicitation  sessions  with  our  SMEs,  observe  patient  therapy, 
and  develop  an  initial  draft  of  the  complete  requirements  for  the  system. 

■  Under  Task  2,  we  will  continue  to  design  the  MOVER  mobile  application  and  identify  needed  extensions 
to  our  existing  in-house  mobile  application  framework. 

■  Under  Task  3,  we  will  begin  significant  design  of  the  virtual  coaches. 

■  Under  Task  4,  we  will  continue  to  investigate  potential  input  devices  and  begin  integration  with  the 
Microsoft  Kinect. 

■  Under  Task  5,  we  will  begin  design  and  development  of  the  remote  server  and  therapist  interface. 


ISSUES  OR  CONCERNS 

During  this  reporting  period,  we  encountered  an  issue  with  potential  funding  of  the  MOVER  Option  (Task  6  in  the 
Statement  of  Work).  The  MOVER  Option  contains  primary  evaluation  task  with  patients.  To  address  this  issue,  we 
will  continue  to  develop  the  system  with  input  from  subject  matter  experts.  We  will  continue  to  seek  possibilities 
for  funding  the  option  and  explore  follow-on  efforts  for  clinical  evaluation. 

We  do  not  foresee  any  other  potential  issues  at  this  time. 
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EXPENDITURES 


Total  Contract  Amount 

$767,388.00 

Costs  Incurred  this  reporting  period 

$39,933.00 

Costs  Incurred  to  Date 

$39,933.00 

Estimated  %  to  completion 

5% 

